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Abstract — Human operators confronted with mis- 
aligned display and control frames of reference per- 
formed three-dimensional, pursuit tracking in virtual 
environment and virtual space simulations. Analysis 
of the components of the tracking errors in the 
perspective displays presenting virtual space 
showed that components of the error due to visual- 
motor misalignment may be linearly separated from 
those associated with the mismatch between display 
and control coordinate systems. Tracking perfor- 
mance improved with several hours practice despite 
previous reports that such improvement did not take 
place. 

I. INTRODUCTION 

The design task for telemanipulation displays 
is complicated by the fact that the three-dimensional 
data they depict may entail viewing from nonoptimal 
directions. The viewing projection is described by a 
number of parameters each of which can substantially 
alter the appearance of the resulting image. Since the 
purpose of the display is the accurate depiction of 
spatial information and successful manipulation of 
objects, designers must first understand the effects of 
these display parameters on operator perception and 
control in the work space. Studies of the effects of 
some of the major projection parameters, i.e. viewing 
direction and field of view angle (FOV), on the per- 
ception of exocentric direction have been reported re- 
cently [1] [2] [3] [4]. Other studies of manipulation and 
control of telemanipulated objects also have been re- 
ported over the past 20 years [5] 16] [71 [8][9] [10] [11]. 

The present experiments and analysis extend 
these investigations with studies of pursuit tracking of 
virtual objects moving irregularly in three dimensions 
and focus on the spatial components of the tracking er- 
ror. In these experiments the FOV angle is defined as 


the visual angle subtended by the viewport as seen 
from the geometric center of the projection. The azi- 
muth and elevation of the viewing vector are defined 
by angles between the world coordinate system and an 
extension of the viewing vector to a reference point. In 
general, the FOV was correctly matched by placing 
the subject's eye at the correct station point while 
misalignments between display and control axes were 
introduced by rotating the azimuth of the viewing 
direction. These misalignments were used to produce a 
decrement in human tracking performance in a manner 
similar to that of studies by Bernotat [8]. The resulting 
pattern of degraded performance, which has been 
found to be a function of the amount of misalignment, 
provides a basis for examining Bemotat’s claim that 
subjects cannot adapt to the misalignment. In addition, 
errors in tracking will provide data to assess the 
suitability of integrated measurement of tracking 
error versus analysis of the components of the error. 

II. EXPERIMENT 1 

A. Three-dimensional Tracking with Perspective 
Display and Joystick Controllers 

B. Methods 

This experimental setup was identical to that 
described in an earlier paper [9]. The basic task was 
to control two, 2-axis joysticks to track the projection 
of a small diamond-shaped target that moved 
irregularly by sums of sines in three dimensions. The 
forcing functions in all three dimensions were 
determined so their spectrum resembled that of a first 
order, low pass filter with a cut-off at 0.1 Hz. Five 
different forcing functions were precomputed and ran- 
domly selected for a particular two minute tracking 
run. Subject's were given a ten second warm-up at the 
outset of each two-minute tracking run. The sampling 
and display frequency was 40.96 Hz. 
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The perspective projection on the screen was 
computed by an 11/23 computer and displayed on a 
HP1345A stroke monitor. The cursor was a small cross 
displayed at half brightness. It was controlled in 
horizontal position by two axes of one joy-stick and in 
vertical position by the fore-aft axis of the other 
joystick. The tracking environment, a 5x5 line grid, 
was oriented to appear parallel to the floor and 
provide a spatial reference for pursuit tracking. Four 
tracking errors were computed: Integrated normalized 
root mean square (RMS) error between the target and 
cursor, and the separate x, y and z components of this 
error. In normalized RMS tracking error a value of 1.0 
corresponds to the error expected if no tracking is at- 
tempted while the cursor remains centered. If t is the 
target position and c is the cursor position and x = 
(x,y,z), the computed errors were: 
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Figure 1. Schematic setup for 3D tracking using two 
2 axis joysticks with a target on a perspective dis- 
play.The control axes of the right joystick could be 
rotated into alignment with the viewing vector. To 
the right of the schematic is a tracking display 
image with a 45° rotation of azimuth and a pitch 
down of 45°. 

The view vector was pitched down 45° and the az- 
imuth was rotated through 9 positions either cw or 


ccw to provide a variable amount of display and 
control misalignment. The central axis of the grid 
represented the X and Z coordinates of a Cartesian 
coordinate system. The experiment was conducted in 
a darkened room but the contours of the display case 
and other laboratory equipment were still easily 
visible and well into photopic vision. Subject’s were 
seated 40 cm in front of the display screen at the 
correct geometric eye point for the projection as shown 
in Figure 1. 

The principal manipulation of the experiment 
was the orientation of the control axes of the joystick 
and the orientation of the viewing vector. In one 
condition it remained constant (unrotated condition) 
as the viewing vector was rotated, thus allowing a 
display-control misalignment to develop. In the sec- 
ond condition (rotated condition) the control axes 
were rotated so as to maintain alignment with the 
display axes as the view vector was rotated (See 
Figures 1 and 3). The comparison of tracking 
performance in these two different conditions would 
allow the investigators to isolate of^that tracking 
component due to the purely visual consequences of ro- 
tating the viewing vector, shown in the rotated 
condition, from the consequences of visual-motor mis- 
alignment. The purely visual-motor component would 
be seen as the difference between the unrotated and 
the rotated conditions. The direction of rotation 
(cw/ccw) was reversed for alternate subjects. Half 
the group experience the rotated joystick axis condi- 
tion first and half the unrotated condition first. The 
specific sequence of rotations presented was 
randomized before each experimental condition was 
run with a particular subject. 

C. Subjects 

Six male subjects enroled in a mechanical engi- 
neering course at U.C. Berkeley served as subjects. 
They had all had sufficient experience with the 
tracking task to have reached asymptotic perfor- 
mance with joy-sticks and the general task, but they 
had had no practice with all the misalignment 
conditions. 

D. Results 

Inspection of the data averaged across subjects 
in Figure 2 for the unrotated, normal conditions shows 
two significant features. First, the best tracking ap- 
pears to be at an azimuth of 0°. There appears to be 
very little effect on tracking performance until the 
azimuth angle exceeds roughly 50°. Thereafter, the 
error increases to a maximum at about 125° thereafter 
decreasing to 180°. Rotation of the control axes to 
maintain visual-motor alignment had appeared to 
eliminate totally the tracking disturbance. However, 
further analysis of the components (Figure 4) of the 
tracking revealed a residual tracking error, at- 
tributable purely to the visual aspects of the track- 
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Figure 2. Comparison of RMS tracking error for 
rotated and unrotated tracking conditions. 


ing. The visual component of the tracking along an 
axis is most accurate if the axis is viewed from an 
orthogonal direction. The variable component of 
tracking is roughly proportional to the sine of the 
angular deviation from the optimal view. Thus, the 
peaks in the x and z components tracking error were 
90° apart. Significantly, if the visual component 
revealed by the rotated condition is subtracted from 
the unrotated condition, the x and z components of the 
tracking are now almost identical. This similarity 
suggests that the visual component and the visual- 
motor component of the tracking errors may be 
linearly separable. 

III. EXPERIMENT 2 

A. Three-Dimensional Tracking in Virtual Environ- 
ment with Anthropomorphic 3-dof controller 

In contrast to Bemotat's report [8] that the perfor- 
mance disruptions caused by display-control mis- 
alignments do not dissipate with practice, we have 
observed in a previous three dimensional tracking 
experiment [2] evidence that tracking does improve 
with several hours practice. To verify that this 
training effect also takes place in other tracking 
environments, a tracking experiment similar to 
Experiment 1 was conducted in a virtual environment 
using a hand position sensor. In this case the subject 
used a direct mapping of hand position to cursor 
position and did not have to learn the motor 
mappings involved in use of a joystick to control a cur- 
sor. The more natural mode of cursor control provided 
by a hand tracker could be expected to facilitate 
adaptation to the display-control misalignment. 

B. Methods 

The earlier tracking studies (2) as well as Ex- 
periment 1 used 5 distinct target disturbance functions 
with the identical frequency content. The target 
functions were randomly selected for each run 
without the subject's knowledge. This second 


experiment which copied the dynamic and vi- 
sualcharacteristics of the previous experiments, used 
unique forcing functions for each run. Use of unique 
functions for each run can rule out the remote pos- 
sibility that subjects can learn the specific paths in 
the fixed target function. 

In the present study the tracking environment 
was presented as a virtual stereoscopic image via a 
head-mounted, CCD (charge-coupled device) 
display developed for technology demonstration pur- 
poses at the Aerospace Human Factors Division at 
the Ames Research Center [12]. For the tracking ex- 
periment the overall sampling and display rate was 
approximately 30 Hz. Other experimental conditions 
were comparable to those in Experiment 1. The 
subject’s head position was tracked electro- 
magnetically during the experiment and used to 
approximately inertially stabilize the visual dis- 
play. They were not, however, encouraged to walk 
about after first selecting a viewpoint that appeared 
subjectively optimal. Since the entire grid display 
was visible without head movement, none was re- 
quired during the tracking. But both subjects made 
small (<20°) rotational movements anyhow. The 
subjects stood directly below a Polhemus electromag- 
netic 6 DOF tracker which was mounted on a plastic 
support 7 ft above the ground. Based on a calibration 
test, the reach volume of both subject’s right hands 
was within a volume in which the tracker performed 
close to factory specifications. The tracking space 
was scaled 1:1 within the synthetic environment and 
the gain for displacement in all axes was set at 1.0. 
The subjects tracked targets under the nine different 
visual-motor misalignment conditions with two 
replications in each set for a total of 18 randomly or- 
dered tracking runs per group. Each subject was given 
3 groups for a total of 54 runs. Subjects were given rest 
period of approximately 10 minutes between groups. 
Before data collection began subjects were given sev- 
eral practice runs and informal training under 
unrotated tracking conditions to be sure they un- 
derstood the tracking task and that the head- 
mounted display was properly positioned. As in Ex- 
periment 1, the first 10 seconds of tracking was con- 
sidered a warm-up period and not analyzed 

C. Subjects 

Two male subjects, ages 18 and 63 years, who were 
laboratory personnel at Ames participated in the 
experiment. 

D. Results 

The individual tracking RMS errors from both 
subjects (Figure 6) indicate a similar performance 
patterns compared to earlier data. As the mis- 
alignment between display and control axes is in- 
creased, tracking generally deteriorated. The rela- 
tive amount of the deterioration seemed to be less and 
maximum degradation around 125° of misalignment is 
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Figure 3. A target at • is disturbed in 3 dimension and tracked by cursor at O . The viewing is with azimuth of 'F and el- 
evation of 0 from COP. Integrated instantaneous tracking error and its x,y,z components are ds, dx,dy, and dz respec- 
tively. Projections of these errors are on the viewplane, as ds’, dx', dy’ and dz' The view plane has been redrawn to the 
right for clarity. The disk of dots on the right view plane represents line of sight measurement noise. 



Figure 4. Across subjects means (n=6) of integrated RMS and its x,y,z components for unrotated (upper traces) and 
rotated control-axes (middle traces) during 3D tracking on perspective displays. The lower trace in each figure results 
from subtracting the rotated conditon from corresponding tracking results in the unrotated condition 
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not as pronounced as in earlier results. Tracking 
performance of the two subjects does not appear to 
stabilize until the third group of tracking runs. Dif- 
ferences in the absolute RMS values reflect dif- 
ferences in individual tracking capabilities as well 
as the poor display resolution and temporal response 
of the head-mounted display. Significantly, despite 
many technical and procedural differences with 
earlier studies, both subjects confirm tracking per- 
formance improvement. In fact, the peak disturbance 
around 125° of misalignment almost vanishes in both 
subjects (Also see Figure 7 from an earlier experiment) 

£. Discussion 

Difference in dynamics used to control the cur- 
sor as compared to Bemotat’s earlier work may ac- 
count for the above confirmation that subjects can 
adapt to the misalignment of the control frame of 
reference . Bernotat simulated a 2nd order system 
which gave the cursor a significant sluggishness 
causing a delay between the subjects actions and his 
observation of an effect. This delay could disturb sub- 
ject's ability to learn a compensatory motor strategy 
that would improve from session to session. Another 
significant difference was that Bernotat used 
compensatory tracking and we used a pursuit 
paradigm. The compensatory tracking may have ob- 
scured the effects of the subject's control inputs by 
mixing them with the targets own motion. This 
mixing increases the difficulty he would have in 
identifying the effects of his control inputs on the 
target. 



Figure 5. Subject wearing the virtual environment dis- 
play while performing 3D tracking. 

A common feature of the several experiments 
we have conducted on display-control misalignment 
has been that the worst tracking performance seems 
to be at at misalignment angle somewhat greater 
that 90°. This finding is even suggested by 
Experiment 2 in that the peak error seems to be at 
rotations greater than 90°, perhaps around 125°. The 


peak may remain evident after considerable practice 
Experiment 1 again confirms this point but mon 
subject's are needed to confirm it in Experiment 2. 




Figure 6 Two Individual subjects integrated RMS 
3D tracking data averaged within each group of 
18 runs. 

The observation that the poorest tracking perfor- 
mance may be at azimuth rotations near 125 ° invites 
speculation. The subjects ability to compensate for 
the viewing rotation of the reference grid with 
respect to the control axes of the joystick may involve 
a mental rotation of the grid. Cooper and Shepard 
[ 13 ] have shown that the time required for such 
mental rotation varies in proportion to the size of 
the angle of rotation. If this effect occurs in manual 
tracking, a phase lag between the target trajectory 
and the human response should be detected that is 
proportional to the azimuth angle used to generate 
each display condition. 

But why should the maximum tracking error occur at 
a rotation greater than 90 °? At 90 ° the subject could 
rotate either clockwise or counterclockwise to align 
with the control axes. Depending upon how the sub- 
ject visualizes his mental rotation, one or the other of 
these directions of rotation will also require a control 
reversal. Clearly, the subjects will choose the rota- 
tion that does not require a reversal. But what should 
they do at 91 °? Assuming that the control reversal 
will add additional time lags and further degrade 
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Figure 7. The results of multiple two minute track- 
ing runs from a single well-practiced subject who 
experience display-control rotational misalign- 
ments from 0 to 180° in an earlier experiment using a 
perspective display [2J. The data show both 
evidence of learning across groups of runs (1,2,3) and 
a peak disturbance in tracking near 125°. 

tracking performance, one may speculate that the 
subject will choose to make mental rotations greater 
than 90° to avoid having to introduce the reversal. 

At some angle, however, a point may be 
reached at which the delay due to a control reversal 
may be acceptable because the alternative rotation 
becomes sufficiently small. Thereafter, the total de- 
lay should be a decreasing function of azimuth; as the 
azimuth increases, the amount of required mental 
rotation decreases. Another consequence of this 
analysis is that the intra and intersubject variability 
ought to be at a maximum near the point of maximum 
RMS error since this would be where most equivo- 
cation regarding the direction of corrective rotation 
should occur. 

Understanding of the consequences and causes of 
spatial interpretation and control of three-di- 
mensional cursors will assist in the design of three- 
dimensional spatial instruments for use in aerospace 
applications i.e. [14]. 

IV. CONCLUSIONS 

1) Manual three dimensional pursuit tracking errors 
produced by display-control rotational mis- 
alignments have two linearly separable components: 
a purely visual component and a visual-motor com- 
ponent. Both components may be independent 
influences on tracking performance. 

2) Human subjects can simultaneously adapt to a 
variety of display-control misalignments if using 
position control during pursuit tracking with a 


simulation update rate of at least 30 Hz. This 
capability will enable trained operators to quickly 
adapt to changes in the position and orientation of 
viewing cameras during teleoperation and 
telemanipulation. 
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